Skip to content

DRIVER-153: negotiate and implement SCYLLA_USE_METADATA_ID extension#770

Open
nikagra wants to merge 8 commits into
scylladb:masterfrom
nikagra:driver-153-scylla-use-metadata-id
Open

DRIVER-153: negotiate and implement SCYLLA_USE_METADATA_ID extension#770
nikagra wants to merge 8 commits into
scylladb:masterfrom
nikagra:driver-153-scylla-use-metadata-id

Conversation

@nikagra
Copy link
Copy Markdown

@nikagra nikagra commented Mar 26, 2026

Summary

Implements the SCYLLA_USE_METADATA_ID Scylla CQL protocol extension (DRIVER-153), which backports the prepared-statement metadata-ID mechanism from CQL v5 to earlier protocol versions.

When the extension is negotiated:

  • The server includes a result metadata hash in the PREPARE response
  • The driver sends that hash back with every EXECUTE request, allowing the server to skip sending full result metadata on every response (skip_meta=True)
  • If the result schema has changed, the server sets the METADATA_CHANGED flag and includes the new metadata ID + new column metadata in the response — the driver picks this up and updates its cached metadata automatically

Changes

cassandra/protocol_features.py

  • Add USE_METADATA_ID = "SCYLLA_USE_METADATA_ID" constant and use_metadata_id field to ProtocolFeatures
  • Parse the extension from the SUPPORTED frame; include it in STARTUP when present

cassandra/protocol.py

  • Bug fix: _write_query_params now actually writes _SKIP_METADATA_FLAG on the wire — it was stored on _QueryMessage but never sent (effectively dead code)
  • recv_results_prepared: read result_metadata_id for Scylla extension (pre-v5) in addition to standard CQL v5+
  • ExecuteMessage: add use_metadata_id flag (default False); send_body gates the result_metadata_id field on ProtocolVersion.uses_prepared_metadata(protocol_version) or self.use_metadata_id — both paths unified. An empty sentinel (b'') is written when the hash is None (LWT / mixed cluster) so the frame layout is always correct and no TypeError crash on v5

cassandra/cluster.py

  • _create_response_future: build ExecuteMessage with safe defaults (skip_meta=False, result_metadata_id=None, use_metadata_id=False)
  • _query: after borrowing the connection, set can_skip_meta, skip_meta, result_metadata_id, and use_metadata_id based on the actual connection's negotiated features. skip_meta is only enabled when the prepared statement has both a result_metadata_id and usable cached result_metadata — guards against LWT/NO_METADATA statements and mixed-cluster scenarios
  • _set_result: on METADATA_CHANGED update prepared_statement.result_metadata then result_metadata_id in that order (safe write ordering for concurrent readers)

docs/scylla-specific.rst

  • Document the extension, its behaviour, and the reference to the ScyllaDB CQL extensions spec

Test plan

  • 19 new unit tests across 3 files covering: feature negotiation, STARTUP options, skip_meta flag encoding, metadata_id in ExecuteMessage (v4/v5, sentinel for None), PREPARE response decoding with/without extension, METADATA_CHANGED and NO_METADATA flag handling, _set_result METADATA_CHANGED path, and per-connection feature gating in _query (5 scenarios: extension on/off, metadata id present/absent, LWT, protocol v5)
  • Full unit test suite passes (57 passed in affected files; 627+ overall)
  • Integration tests against a Scylla node with the extension: verify that schema changes after PREPARE are detected and metadata is updated without re-preparation

Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Implements negotiation and support for Scylla’s SCYLLA_USE_METADATA_ID protocol extension to enable metadata-id based skip_meta behavior (backporting CQL v5 prepared-statement metadata-id semantics to earlier protocol versions).

Changes:

  • Adds SCYLLA_USE_METADATA_ID parsing from SUPPORTED and includes it in STARTUP when negotiated.
  • Extends protocol encode/decode to read/write result_metadata_id for PREPARE/EXECUTE on pre-v5 when the extension is used, and fixes on-wire encoding of _SKIP_METADATA_FLAG.
  • Updates execution/result handling to conditionally use skip_meta and to refresh cached prepared metadata when the server reports metadata changes.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
cassandra/protocol_features.py Adds the SCYLLA_USE_METADATA_ID feature flag and includes it in negotiated STARTUP options.
cassandra/protocol.py Writes _SKIP_METADATA_FLAG in query params; adds pre-v5 extension handling for result_metadata_id in PREPARE/EXECUTE.
cassandra/cluster.py Adjusts when skip_meta is enabled and updates cached prepared metadata/id on METADATA_CHANGED responses.
tests/unit/test_protocol_features.py Adds unit tests for feature parsing and STARTUP option inclusion.
tests/unit/test_protocol.py Adds unit tests for skip-meta flag encoding and metadata-id handling in pre-v5 PREPARE/EXECUTE paths.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread cassandra/cluster.py Outdated
Comment thread cassandra/protocol.py Outdated
Comment on lines +648 to +649
elif self.result_metadata_id is not None:
write_string(f, self.result_metadata_id)
Copy link

Copilot AI Mar 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ExecuteMessage.send_body() now writes result_metadata_id for protocol versions that don’t use standard prepared-metadata (pre-v5) whenever self.result_metadata_id is non-None. For pre-v5 this extra field is only valid when SCYLLA_USE_METADATA_ID was negotiated on that connection; otherwise it changes the wire layout (the server will interpret the metadata id bytes as the query parameters) and the request will fail.

To make this safe, ensure the decision to include result_metadata_id is gated by the negotiated feature (e.g., add an explicit use_metadata_id/send_result_metadata_id flag on the message that the caller sets based on connection.features.use_metadata_id, or guarantee centrally that result_metadata_id is cleared unless the extension is active for that connection).

Suggested change
elif self.result_metadata_id is not None:
write_string(f, self.result_metadata_id)

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. The elif self.result_metadata_id is not None path in send_body is now only reached when the caller explicitly set the field — which only happens in _query() after confirming connection.features.use_metadata_id (or CQL v5). For any connection that didn't negotiate the extension, result_metadata_id remains None and the branch is never taken, so the wire layout is unaffected.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is still a problem here. If the result id feature is negotiated on the connection, then you need to ALWAYS send some result metadata id in EXECUTE. Skipping the write_string will result in a protocol error.
Your use_metadata_id may be False even if extension was negotiated, if the server decided to skip the metadata in PREPARED response. In such case, you'll skip writing the id here, and encounter protocol error.

Even if you fix this specific case, there is still possibility of mixed cluster, with some nodes supporting the extension. In that case result_metadata_id will be None, and if you send to a node that has the extension negotiated, you'll again not send the id and encounter protocol error.

To sum up: this serialization here should check if feature is negotiated, and base sending this field only on that.

Copy link
Copy Markdown
Author

@nikagra nikagra May 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👨‍💻: Addressed your concerns

🤖:

Automated response

Fixed. Added a `use_metadata_id` boolean flag to `ExecuteMessage` (default `False`). `send_body` now gates the `result_metadata_id` field on `ProtocolVersion.uses_prepared_metadata(protocol_version) or self.use_metadata_id` — both branches unified — writing an empty sentinel (`b''`) when the hash is `None` rather than omitting the field or crashing.

In _query(), message.use_metadata_id = connection.features.use_metadata_id is set unconditionally after borrowing the connection, so the decision is always based on what was negotiated on that specific connection.

This also fixes a second issue: the old separate v5 branch would crash with TypeError when result_metadata_id is None (LWT statement on v5, or statement prepared before the extension was active in a mixed cluster). The unified sentinel handles that case correctly on both code paths.

Two new tests cover this: test_execute_message_scylla_metadata_id_none_writes_sentinel (extension active, None hash → sentinel on v4) and test_execute_message_v5_metadata_id_none_writes_sentinel (v5 native path, None hash → sentinel instead of crash).

Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread cassandra/cluster.py Outdated
@nikagra nikagra force-pushed the driver-153-scylla-use-metadata-id branch from ade35d8 to f42e225 Compare March 27, 2026 12:32
@mykaul
Copy link
Copy Markdown

mykaul commented Mar 29, 2026

I'm not sure where, but we should document this - with reference mainly to the scylladb docs about this feature.

@nikagra
Copy link
Copy Markdown
Author

nikagra commented Mar 30, 2026

@mykaul Documentation I'm aware of is MetadataId extension in CQLv4 Requirement Document

@nikagra nikagra requested a review from sylwiaszunejko April 9, 2026 11:12
@nikagra nikagra marked this pull request as ready for review April 9, 2026 21:30
Copy link
Copy Markdown
Collaborator

@dkropachev dkropachev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One blocking correctness issue below: skip_meta is being enabled for prepared statements that can still have empty/absent cached result metadata.

Comment thread cassandra/cluster.py Outdated
@nikagra nikagra force-pushed the driver-153-scylla-use-metadata-id branch from 6eea397 to a86fd53 Compare April 15, 2026 09:09
@nikagra nikagra requested a review from dkropachev April 15, 2026 09:12
@nikagra nikagra force-pushed the driver-153-scylla-use-metadata-id branch from 7ba5835 to a86fd53 Compare April 15, 2026 11:12
@nikagra nikagra force-pushed the driver-153-scylla-use-metadata-id branch from a86fd53 to 8880f03 Compare April 22, 2026 12:34
@nikagra nikagra force-pushed the driver-153-scylla-use-metadata-id branch from 170fd31 to 5fe1902 Compare May 14, 2026 14:45
@nikagra nikagra requested a review from Lorak-mmk May 14, 2026 14:45
@nikagra nikagra force-pushed the driver-153-scylla-use-metadata-id branch from fcd3eba to 5fe1902 Compare May 15, 2026 09:22
@nikagra nikagra force-pushed the driver-153-scylla-use-metadata-id branch from 5fe1902 to 251b1a8 Compare May 28, 2026 20:32
@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented May 28, 2026

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 3ea2c562-f20a-45d5-a6ac-c95ad23d0fee

📥 Commits

Reviewing files that changed from the base of the PR and between 251b1a8 and de8d3fc.

📒 Files selected for processing (7)
  • cassandra/cluster.py
  • cassandra/protocol.py
  • cassandra/protocol_features.py
  • docs/scylla-specific.rst
  • tests/unit/test_protocol.py
  • tests/unit/test_protocol_features.py
  • tests/unit/test_response_future.py
✅ Files skipped from review due to trivial changes (1)
  • docs/scylla-specific.rst
🚧 Files skipped from review as they are similar to previous changes (6)
  • tests/unit/test_protocol_features.py
  • cassandra/protocol.py
  • cassandra/protocol_features.py
  • tests/unit/test_protocol.py
  • cassandra/cluster.py
  • tests/unit/test_response_future.py

📝 Walkthrough

Walkthrough

This PR adds Scylla's SCYLLA_USE_METADATA_ID support: negotiates a new protocol feature, encodes/decodes result_metadata_id and skip_meta/use_metadata_id in QUERY/EXECUTE/PREPARE/RESULT messages, defers attaching metadata-id info until a connection is borrowed, and updates prepared-statement cached metadata when servers send new result_metadata_id. It includes docs and unit tests covering v4/v5 and the Scylla extension paths.

Sequence Diagram

sequenceDiagram
  participant Client
  participant Connection
  participant Server
  Client->>Connection: execute(prepared_statement)
  activate Connection
  Connection->>Connection: ResponseFuture._query(): check cached metadata<br/>& connection features
  alt has cached metadata & supports metadata-id
    Connection->>Server: ExecuteMessage(skip_meta=True, result_metadata_id=cached_id)
  else no cached metadata or unsupported
    Connection->>Server: ExecuteMessage(skip_meta=False, result_metadata_id=None)
  end
  activate Server
  alt metadata unchanged
    Server->>Connection: ResultMessage(result_metadata_id=cached_id, no column_metadata)
  else metadata changed
    Server->>Connection: ResultMessage(result_metadata_id=new_id, column_metadata=[...])
  end
  deactivate Server
  Connection->>Connection: ResponseFuture._set_result(): if result_metadata_id present,<br/>update prepared_statement metadata cache
  deactivate Connection
  Connection->>Client: return rows with cached metadata
Loading
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 65.96% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly identifies the main change: negotiating and implementing the SCYLLA_USE_METADATA_ID extension, which is the core objective of this PR.
Description check ✅ Passed The description covers all required sections: detailed summary of changes, per-file implementation details, comprehensive test coverage, and appropriate documentation references. It addresses the template checklist items thoroughly.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.


Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented May 28, 2026

Actionable comments posted: 0

nikagra added 5 commits June 1, 2026 17:50
Backport the prepared-statement metadata-ID mechanism from CQL v5 to
earlier protocol versions via the SCYLLA_USE_METADATA_ID Scylla extension.

protocol_features.py:
- Add USE_METADATA_ID constant and use_metadata_id field to ProtocolFeatures
- parse_from_supported: detect SCYLLA_USE_METADATA_ID in SUPPORTED options
- add_startup_options: echo the extension back in STARTUP when negotiated

protocol.py:
- _write_query_params: fix _SKIP_METADATA_FLAG — it was stored but never
  written on the wire (dead code); now correctly sets the flag in the frame
- recv_results_prepared: read result_metadata_id when the Scylla extension
  is active (pre-v5), in addition to the existing CQL v5 path
- ExecuteMessage: add use_metadata_id flag (default False); send_body gates
  the result_metadata_id field on uses_prepared_metadata(protocol_version)
  OR use_metadata_id, writing an empty sentinel (b'') when the hash is
  unavailable (LWT / mixed cluster) instead of omitting the field entirely
  or crashing with TypeError

cluster.py:
- _create_response_future: build ExecuteMessage with defaults (skip_meta=False,
  result_metadata_id=None); set these in _query() once the connection is known
- _query: after borrowing the connection set can_skip_meta, skip_meta,
  result_metadata_id, and use_metadata_id on the ExecuteMessage based on the
  actual connection's negotiated features; skip_meta is only enabled when the
  prepared statement has both a result_metadata_id AND cached result_metadata
  (guards against LWT/NO_METADATA statements and mixed-cluster re-prepare)
- _set_result: on METADATA_CHANGED (new result_metadata_id in EXECUTE response)
  update prepared_statement.result_metadata then result_metadata_id in that
  order — safe write ordering so a concurrent reader using the old id gets
  full metadata from the server while a reader seeing the new id has the
  correct cached metadata immediately

docs/scylla-specific.rst: document the extension and its behaviour
test_protocol_features.py:
- test_use_metadata_id_parsing: SCYLLA_USE_METADATA_ID parsed from SUPPORTED
- test_use_metadata_id_missing: use_metadata_id False when key absent
- test_use_metadata_id_startup_options: key present in STARTUP when negotiated
- test_use_metadata_id_not_in_startup_when_not_negotiated: absent otherwise

test_protocol.py:
- test_execute_message_skip_meta_flag: _SKIP_METADATA_FLAG (0x02) is written
- test_execute_message_scylla_metadata_id_v4: result_metadata_id written on
  v4 when use_metadata_id=True (Scylla extension)
- test_execute_message_scylla_metadata_id_none_writes_sentinel: extension
  active but result_metadata_id=None writes empty sentinel b'' (LWT / mixed
  cluster); frame layout preserved
- test_execute_message_v5_metadata_id_none_writes_sentinel: v5 with
  result_metadata_id=None writes empty sentinel instead of TypeError crash
- test_recv_results_prepared_scylla_extension_reads_metadata_id
- test_recv_results_prepared_no_extension_skips_metadata_id
- test_recv_results_metadata_changed_flag
- test_recv_results_metadata_no_metadata_flag_skips_metadata_id

test_response_future.py:
_set_result METADATA_CHANGED path:
- test_set_result_updates_metadata_when_metadata_changed
- test_set_result_does_not_update_metadata_when_metadata_id_absent
- test_set_result_warns_when_metadata_id_but_no_column_metadata

_query per-connection feature gating (5 scenarios):
- test_query_sets_skip_meta_with_scylla_extension
- test_query_no_skip_meta_without_extension
- test_query_no_skip_meta_when_prepared_statement_has_no_metadata_id
- test_query_sets_skip_meta_for_protocol_v5
- test_query_no_skip_meta_when_result_metadata_is_none (LWT guard)
The test used ps.result_metadata = [] (falsy), making has_result_metadata=False
the reason can_skip_meta was False — not the use_metadata_id=False guard that
the test claims to exercise.  A connection with use_metadata_id=True but empty
result_metadata would have passed the same assertions, giving false confidence.

Use truthy result_metadata so that use_metadata_id=False is the sole deciding
factor, which is what the docstring says the test is verifying.
… type

Issues addressed:
- test_repeat_orig_query_after_succesful_reprepare used result_metadata_id='foo'
  (str) where the real type from read_binary_string is always bytes.  Changed to
  b'foo' and added assertion that the value is actually stored on the prepared
  statement (previously, deleting the update lines would not break any test).

New tests:
- test_execute_after_prepare_updates_result_metadata_id: verifies that after a
  successful reprepare both result_metadata and result_metadata_id are updated
  on the prepared statement.
- test_execute_after_prepare_no_metadata_id_in_response: verifies that when the
  reprepare response carries no result_metadata_id (extension not active on the
  reprepare connection), the existing result_metadata_id is left unchanged.
…ag_skips_metadata_id

The previous assertion used 'not hasattr(...) or ... is None' which would pass
even if the code had a bug that explicitly set result_metadata_id = None instead
of leaving the attribute absent.

result_metadata_id is not a class attribute on ResultMessage, so the correct
assertion is 'assert not hasattr(msg, "result_metadata_id")'.

column_metadata IS a class attribute (= None), so its assertion is adjusted to
'assert msg.column_metadata is None'.
nikagra added 3 commits June 1, 2026 17:54
Both existing recv_results_prepared tests used protocol_version=4, leaving
the ProtocolVersion.uses_prepared_metadata() branch (v5+) untested.

The new test confirms that on protocol v5, result_metadata_id is read from
the PREPARE response even when use_metadata_id=False (native v5 path).
Before this PR, _SKIP_METADATA_FLAG was never written to the wire — the skip_meta
field existed on _QueryMessage but _write_query_params did not use it.  The PR
activates it for the first time for all protocol versions.

The new test pins the v5 wire encoding: skip_meta=True must produce a 4-byte
flags word of 0x00000003 (VALUES_FLAG | SKIP_METADATA_FLAG), alongside the
always-present result_metadata_id sentinel for v5.
…path

The existing test used column_metadata=[] (falsy empty list).  The warning
branch also triggers for None (attribute absent entirely), which has different
semantics — it indicates the field was never set rather than explicitly empty.

The new test ensures both falsy forms trigger the warning and update
result_metadata_id without touching result_metadata.
@nikagra nikagra force-pushed the driver-153-scylla-use-metadata-id branch from 251b1a8 to de8d3fc Compare June 1, 2026 15:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants